Reinforcement Control via Heuristic Dynamic Programming

Heuristic Dynamic Programming (HDP) is the simplest kind of Adaptive Critic which is a powerful form of reinforcement control 1]. It can be used to maximize or minimize any utility function, such as total energy or trajectory error, of a system over time in a noisy environment. Unlike supervised learning, adaptive critic design does not require the desired control signals be known. Instead, feedback is obtained based on a critic network which learns the relationship between a set of control signals and the corresponding strategic utility function. It is an approximation of dynamic programming 2]. A simple Heuristic Dynamic Programing (HDP) system involves two subnetworks, the Action network and the Critic network. Each of these networks includes a feedforward and a feedback component. A ow chart for the interaction of these components is included. To further illustrate the algorithm, we use HDP for the control of a simple, 2-D planar robot.